Implement support for closed TypedDicts (PEP 728)#21382
Conversation
bd9fee9 to
34df265
Compare
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
I'm not sure how exactly you'd prefer to have this change raised, so I've initially opened one big PR containing multiple small commits. Please LMK how you'd like it split up if that's the preference! |
|
First of all thanks for working on this!
This may depend on who exactly will review this PR :-) Although this is a big PR, after quick look it seems quite non-controversial, so the review will be mostly double-checking all edge cases are handled correctly/consistently. So IMO this doesn't need splitting into parts. @JukkaL @hauntsaninja will any of you have time to look at this? (I am not sure if I will have time/energy to do a proper review, as I still have a backlog of NumPy-related stuff.) |
PEP 728 allows multiple special forms to be applied to a key when a TypedDict is created using functional syntax, but the logic was only checking and stripping a single instance. This prevents overriding the overall total status on a ReadOnly field.
Fix a bug where a non-required key was considered consistent with a required key if the latter was marked as read-only.
Fix a bug where the meet of a mutable and readonly key was incorrectly set as readonly. This bug was caused by mistakenly unioning the readonly key sets, which happened to work for the tested case where readonly keys never appeared in the other type.
The TypedDict meet is returning Never whenever inputs have mismatched keys (value types or requiredness). However: * when a key is readonly, the meet value type can be a subtype of it * when a key is readonly and not required, the meet key can be required * when both keys are readonly and not required, the meet value type can be uninhabited (absent)
A typo was causing the readonly state of keys on the LHS of a TypedDict join to be ignored.
Improve TypedDict joins where keys mismatch by returning a readonly key with the joined type instead of discarding it.
Prior to implementing support for subclass refinement, expand the test suite with all the cases that need to be tested.
Allow readonly keys to be refined in subclasses, in line with PEP 705. For cases where refinement is not permitted by spec, provide a more detailed error message. As placeholders may now affect validation without ending up in the final TypedDictType, an `analysis_incomplete` field has been added, triggering reanalysis in subsequent rounds. This closes python#7435
The fallback path in TypeAnalyser::visit_typeddict_type is copying the required keys from the original type, but not the readonly keys. This is resulting in incorrect subtyping analysis for type variables with TypedDict bounds with readonly keys.
`NotRequired[Never]` can be used to indicate that a single key in a TypedDict will not be present.
Begin implementing PEP 728 support by adding an is_closed field to TypedDictType. This is filled out from the closed keyword, and displayed in reveal_type, but not otherwise supported yet.
Continue implementing PEP 728 support with updates to the subtyping logic. Drop the previous use of 'names_are_wider_than', which will be difficult to extend to cover `extra_items`, and instead use zipall to check for key addition/removal alongside the other key-based checks.
Propagate closed from subclasses. Verify that a subclass of a closed TypedDict does not add keys, nor override the closed status.
Close the join of two closed TypedDicts, and treat a missing key in a closed TypedDict as a `NotRequired[Never]` rather than the `ReadOnly[NotRequired[object]]` of an open TypedDict.
The meet of a closed TypedDict and another TypedDict must be closed. The implicit type of a missing key in a closed TypedDict is `ReadOnly[NotRequired[Never]]`.
Narrowing a union already treats final TypedDicts as closed; use the same logic if the closed keyword is used.
If a TypeVar with a TypedDict upper bound is encountered while narrowing, narrow based on the upper bound.
|
I just realised in the course of making these fixes that I made a fundamental misread of PEP 728. It says:
I've implemented it in some places as |
|
My thinking was that you can't delete keys that don't exist, so ReadOnly-ness doesn't make a difference here. Given this program: from typing_extensions import TypedDict
class TD(TypedDict, closed=True):
a: int
def f(x: TD, key: str):
del x["b"]
del x[key]Among type checkers that support PEP 728:
The last behavior makes the most sense to me. |
This comment has been minimized.
This comment has been minimized.
I think I agree, but I fear that results in a weird hybrid of how readonly vs mutable normally works. For example, pop with a default should be allowed in both cases, even though it looks like a mutate operation. I'm just going to go with the literal text of the PEP where it's explicitly called out in the code for now. |
This comment has been minimized.
This comment has been minimized.
271f3e8 to
6734010
Compare
Selecting one of the two join types arbitrarily when they are equivalent does not work when the other one is an `Any` (which is always equivalent to any other type). Instead, recursively join_types in the typeddict join logic in all cases except a missing key in an open type (where we cannot obtain the instance of `object` that would make it clean to do so). This also makes the logic more readable.
Selecting one of the two meet types arbitrarily when they are equivalent does not work when the other one is an `Any` (which is always equivalent to any other type). Instead, recursively meet_types in the typeddict meet logic in all cases except a missing key in an open type (where we cannot obtain the instance of `object` that would make it clean to do so). This also makes the logic more readable.
Compute requiredness and mutability in zipall, and substitute Never for None for closed TypedDicts. This avoids duplicating this logic at the three callsites.
According to PEP 728: > `closed=True`...is equivalent to `extra_items=Never` This should not in practice matter, due to the `Never` type: update operations cannot provide a valid value to insert, and the item must be absent at runtime to conform to the type, so delete operations will have no effect. However, this was not explicitly called out in the PEP, and there may be edge cases that need discussion (e.g. is it legal to pop with default if the item is readonly?). Correct the code to follow the convention exactly.
A closed subtype can validly add a key, provided it is uninhabited, as this is just explicitly stating the previously implicit constraint.
Refer to deletion in error messages for incompatibility mutable not-required fields in base classes, as this is simpler, and ultimately the reason why there is an incompatibility
Calling is_subtype during semantic analysis is not legal; verifying type compatibility of a TypedDict subtype field definition with its supertypes needs to be done in TypeChecker. To avoid duplicating all the logic for determining sources, add a non-persisted field_constraints field to TypedDictType.
6734010 to
2d1333c
Compare
This comment has been minimized.
This comment has been minimized.
|
I accidentally force-pushed a complete rebase 🤦♀️ sorry, I think I managed to revert |
This comment has been minimized.
This comment has been minimized.
ilevkivskyi
left a comment
There was a problem hiding this comment.
Thanks for updates! Couple more comments (in addition to the typeddict_bases comment above).
| readonly_keys.update(base_typed_dict.readonly_keys) | ||
| return info, valid_items | ||
|
|
||
| def field_sources_in_reverse_mro( |
There was a problem hiding this comment.
This name is misleading, since we are only checking compatibility with immediate bases. And to be clear this is the right thing to do, since TypeDicts are structural and are constructed eagerly (i.e. they don't have an MRO).
| class A(TypedDict): | ||
| key: ReadOnly["X"] | ||
| class B(A): | ||
| key: "Y" # ok: Y subclasses X via YB |
There was a problem hiding this comment.
A random idea to make sure that our errors about incompatible overrides are consistent with TypedDict subtyping is to add assignments checks in (some of) those test cases where there are no errors. For example, here you may want to add:
x: A
y: B
x = y # Should be OK as well
ilevkivskyi
left a comment
There was a problem hiding this comment.
Few more comments (as I promised), just some minor things.
| defn.info | ||
| and defn.info.typeddict_type | ||
| and not has_placeholder(defn.info.typeddict_type) | ||
| and not defn.info.typeddict_type.analysis_incomplete |
There was a problem hiding this comment.
Using a flag instead of an extra has_placeholder() check(s) is a good idea, but same thing I mentioned before applies here, I would rather store this information on the TypeInfo. For example, something like info.typeddict_data.ready.
| ): | ||
| typeddict_type.analysis_incomplete = True | ||
| self.api.process_placeholder( | ||
| None, "TypedDict item", info, force_progress=typeddict_type != info.typeddict_type |
There was a problem hiding this comment.
I think force_progress should be updated as well, to match the logic few lines above.
| def primary_source(self, sources: list[FieldSource]) -> FieldSource: | ||
| """Select a primary source from a reverse-MRO-ordered list of sources. | ||
|
|
||
| The primary source will be the last in the MRO list, skipping readonly |
There was a problem hiding this comment.
Just noticed another place where you mention MRO. Instead say reverse order of immediate bases (or something similar).
| key in possible_iterable_type.items or not possible_iterable_type.is_final | ||
| ): | ||
| key in bound.items | ||
| and not isinstance(get_proper_type(bound.items[key]), UninhabitedType) |
There was a problem hiding this comment.
It would be good to add a reference to specific test cases in a comment here.
Despite the name, `TypeInfo` is a symbol (i.e. `SymbolNode`), while `TypedDictType` is a `ProperType`. 10-100x more types than symbols are created when type-checking typical code, so types should be kept "lightweight", as compared to symbols (when possible). Move the information needed to validate TypedDict field type inheritance to a new `typeddict_data` field on `TypeInfo`. Defer creating error messages to `TypeChecker` to avoid storing them unnecessarily.
We are only looking at immediate bases, so 'MRO' is misleading. Revert to previous framing as just "reverse order".
Make sure that our errors about incompatible overrides are consistent with TypedDict subtyping by adding assignment checks in some of the test cases where there are no errors.
|
Diff from mypy_primer, showing the effect of this PR on open source code: steam.py (https://github.com/Gobot1234/steam.py)
- steam/types/trade.py:69: error: Overwriting TypedDict field "instanceid" while merging [misc]
- steam/types/trade.py:69: error: Overwriting TypedDict field "classid" while merging [misc]
- steam/types/trade.py:112: error: Overwriting TypedDict field "assetid" while merging [misc]
- steam/types/trade.py:112: error: Overwriting TypedDict field "amount" while merging [misc]
- steam/types/trade.py:112: error: Overwriting TypedDict field "appid" while merging [misc]
- steam/types/trade.py:112: error: Overwriting TypedDict field "contextid" while merging [misc]
- steam/types/trade.py:112: error: Overwriting TypedDict field "instanceid" while merging [misc]
- steam/types/trade.py:112: error: Overwriting TypedDict field "classid" while merging [misc]
- steam/types/trade.py:112: error: Overwriting TypedDict field "missing" while merging [misc]
altair (https://github.com/vega/altair)
+ altair/vegalite/v6/schema/_config.py:6593: error: Unused "type: ignore" comment [unused-ignore]
+ altair/vegalite/v6/schema/_config.py:6618: error: Unused "type: ignore" comment [unused-ignore]
+ altair/vegalite/v6/api.py:699: error: Unused "type: ignore" comment [unused-ignore]
+ altair/vegalite/v6/api.py:725: error: Unused "type: ignore" comment [unused-ignore]
+ altair/vegalite/v6/api.py:774: error: Unused "type: ignore" comment [unused-ignore]
hydra-zen (https://github.com/mit-ll-responsible-ai/hydra-zen)
- src/hydra_zen/typing/_implementations.py:606: error: Overwriting TypedDict field "module" while extending [misc]
discord.py (https://github.com/Rapptz/discord.py)
- discord/types/emoji.py:42: error: Overwriting TypedDict field "animated" while extending [misc]
+ discord/types/scheduled_event.py:84: error: Field "user_count" is required in base class "_WithUserCount" but can be deleted in base class "StageInstanceScheduledEvent" [misc]
+ discord/types/scheduled_event.py:87: error: Field "user_count" is required in base class "_WithUserCount" but can be deleted in base class "VoiceScheduledEvent" [misc]
+ discord/types/scheduled_event.py:90: error: Field "user_count" is required in base class "_WithUserCount" but can be deleted in base class "ExternalScheduledEvent" [misc]
+ discord/types/interactions.py:214: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/interactions.py:220: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/interactions.py:226: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/interactions.py:232: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/interactions.py:234: error: Field "id" can be deleted in base class "ComponentBase" [misc]
+ discord/types/interactions.py:239: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/interactions.py:241: error: Field "id" can be deleted in base class "ComponentBase" [misc]
+ discord/types/interactions.py:246: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/interactions.py:248: error: Field "id" can be deleted in base class "ComponentBase" [misc]
+ discord/types/interactions.py:268: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/interactions.py:273: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:54: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:59: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:87: error: Definition of field "type" incompatible with base class "SelectComponent" [misc]
+ discord/types/components.py:92: error: Definition of field "type" incompatible with base class "SelectComponent" [misc]
+ discord/types/components.py:97: error: Definition of field "type" incompatible with base class "SelectComponent" [misc]
+ discord/types/components.py:102: error: Definition of field "type" incompatible with base class "SelectComponent" [misc]
+ discord/types/components.py:107: error: Definition of field "type" incompatible with base class "SelectComponent" [misc]
+ discord/types/components.py:113: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:125: error: Definition of field "type" incompatible with base class "SelectComponent" [misc]
+ discord/types/components.py:133: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:139: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:156: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:169: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:174: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:182: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:188: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:195: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:202: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:210: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:220: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
+ discord/types/components.py:232: error: Definition of field "type" incompatible with base class "ComponentBase" [misc]
- discord/types/scheduled_event.py:84: error: Overwriting TypedDict field "user_count" while merging [misc]
- discord/types/scheduled_event.py:87: error: Overwriting TypedDict field "user_count" while merging [misc]
- discord/types/scheduled_event.py:90: error: Overwriting TypedDict field "user_count" while merging [misc]
- discord/types/interactions.py:214: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/interactions.py:220: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/interactions.py:226: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/interactions.py:232: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/interactions.py:234: error: Overwriting TypedDict field "id" while extending [misc]
- discord/types/interactions.py:239: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/interactions.py:241: error: Overwriting TypedDict field "id" while extending [misc]
- discord/types/interactions.py:246: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/interactions.py:248: error: Overwriting TypedDict field "id" while extending [misc]
- discord/types/interactions.py:268: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/interactions.py:273: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/guild.py:142: error: Overwriting TypedDict field "stickers" while extending [misc]
- discord/types/components.py:54: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:59: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:87: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:92: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:97: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:102: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:107: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:113: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:125: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:133: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:139: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:156: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:169: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:174: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:182: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:188: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:195: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:202: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:210: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:220: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/components.py:232: error: Overwriting TypedDict field "type" while extending [misc]
- discord/types/gateway.py:180: error: Overwriting TypedDict field "newly_created" while extending [misc]
- discord/channel.py:144: error: Overwriting TypedDict field "bitrate" while extending [misc]
- discord/channel.py:145: error: Overwriting TypedDict field "user_limit" while extending [misc]
- discord/channel.py:146: error: Overwriting TypedDict field "rtc_region" while extending [misc]
- discord/channel.py:147: error: Overwriting TypedDict field "video_quality_mode" while extending [misc]
- discord/channel.py:148: error: Overwriting TypedDict field "overwrites" while extending [misc]
- discord/channel.py:151: error: Overwriting TypedDict field "topic" while extending [misc]
- discord/channel.py:152: error: Overwriting TypedDict field "slowmode_delay" while extending [misc]
- discord/channel.py:153: error: Overwriting TypedDict field "nsfw" while extending [misc]
- discord/channel.py:154: error: Overwriting TypedDict field "overwrites" while extending [misc]
- discord/channel.py:155: error: Overwriting TypedDict field "default_auto_archive_duration" while extending [misc]
- discord/channel.py:156: error: Overwriting TypedDict field "default_thread_slowmode_delay" while extending [misc]
+ discord/ext/commands/hybrid.py:61: error: Definition of field "description" incompatible with base class "_HybridCommandKwargs" [misc]
+ discord/ext/commands/hybrid.py:69: error: Definition of field "description" incompatible with base class "_HybridCommandDecoratorKwargs" [misc]
+ discord/ext/commands/hybrid.py:73: error: Definition of field "description" incompatible with base class "_HybridGroupKwargs" [misc]
- discord/ext/commands/hybrid.py:61: error: Overwriting TypedDict field "description" while extending [misc]
- discord/ext/commands/hybrid.py:64: error: Overwriting TypedDict field "with_app_command" while extending [misc]
- discord/ext/commands/hybrid.py:65: error: Overwriting TypedDict field "guild_ids" while extending [misc]
- discord/ext/commands/hybrid.py:66: error: Overwriting TypedDict field "guild_only" while extending [misc]
- discord/ext/commands/hybrid.py:67: error: Overwriting TypedDict field "default_permissions" while extending [misc]
- discord/ext/commands/hybrid.py:68: error: Overwriting TypedDict field "nsfw" while extending [misc]
- discord/ext/commands/hybrid.py:69: error: Overwriting TypedDict field "description" while extending [misc]
- discord/ext/commands/hybrid.py:73: error: Overwriting TypedDict field "description" while extending [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "max_messages" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "proxy" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "proxy_auth" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "shard_id" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "shard_count" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "application_id" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "member_cache_flags" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "chunk_guilds_at_startup" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "status" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "activity" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "allowed_mentions" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "heartbeat_timeout" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "guild_ready_timeout" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "assume_unsync_clock" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "enable_debug_events" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "enable_raw_presences" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "http_trace" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "max_ratelimit_timeout" while merging [misc]
- discord/ext/commands/bot.py:96: error: Overwriting TypedDict field "connector" while merging [misc]
pydantic (https://github.com/pydantic/pydantic)
- pydantic/fields.py:99: error: Unexpected keyword argument "closed" for "__init_subclass__" of "TypedDict" [call-arg]
|
Implement support for the closed keyword on TypedDicts (part of PEP 728).
Additionally, fix some preexisting issues that I came across while updating the logic.
Overwriting TypedDict field "x" while merging#8714